WCSAC: Worst-Case Soft Actor Critic for Safety-Constrained Reinforcement Learning

Safe exploration is regarded as a key priority area for reinforcement learning research. With separate reward and safety signals, it natural to cast constrained learning, where expected long-term costs of policies are constrained. However, can be hazardous set constraints on the signal without considering tail distribution. For instance, in safety-critical domains, worst-case analysis required avoid disastrous results. We present novel algorithm called Worst-Case Soft Actor Critic, which extends Critic with critic achieve risk control. More specifically, certain level conditional Value-at-Risk from distribution measure judge constraint satisfaction, guides change adaptive weights trade-off between safety. As result, we optimize under premise that their performance satisfies constraints. The empirical shows our attains better control compared expectation-based methods.